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METHOD AND APPARATUS FOR REDUCING IMAGE 
ACQUISITION TIME IN A DIGITAL IMAGING DEVICE 



Technical Field 

The present invention relates to digital image processing. More 
particularly, the present invention is directed to a method and apparatus for 
reducing the time required to process and store an image on a storage medium 
in a digital imaging device. 

Background of the Invention 

In today's digital world, digital imaging devices, such as digital cameras, 
are enjoying great success. While digital imaging devices may provide superior 
image quality and versatility to the user, the rate with which an image is 
acquired in these devices is often slow. This is usually due to image processing 
operations performed by the device prior to storing the image on a storage 
medium. 

More particularly, in digital imaging devices, an image is captured by an 
image sensor and the raw image data is supplied to a central processing unit 
(CPU) for processing. The processing usually includes conventional image 
processing and compression of the image data. The processed image data is 
then stored onto a storage medium. A significant period of time is often 
required to perform the image processing. More particularly, each part of the 
image processing pipeline is usually performed sequentially on the entire 
image. The processed image data is then stored onto a storage medium. 
Writing the image data for the entire image is also a time-consuming process, 
often much greater than the image processing. In addition, many digital 
imaging devices are unusable during the time from when the image is obtained 
by an image sensor until it is stored on the storage medium. Therefore, digital 
imaging devices do not efficiently support the rapid capture of a series of 
images. 
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In an effort to address this problem, digital imaging devices have been 
developed that delay image processing until some time after the raw image data 
has been stored. More particularly, these devices include a storage medium 
that is usually large enough to store the raw data for up to 10 images. The raw 
image data is stored on the storage medium, and the CPU processes the raw 
data at a later time. However, this approach continues to suffer from the 
disadvantage of limited ability to support the rapid capture of a series of 
images, and suffers from the additional disadvantage of requiring a large 
amount of memory. 

Other devices buffer images that have already been processed, which 
also requires a large amount of memory. In addition, this technique suffers 
from the problem of risking picture loss as the imaging device reports "ready" 
before the image has been completely written to the permanent storage device. 

Therefore, there is a need for a method and apparatus that provides for 
decreased image acquisition time in digital imaging devices. More particularly, 
there is a need for a method and apparatus that enables a digital imaging 
device to capture multiple images rapidly by decreasing the time period 
between capturing a first image and being prepared to capture another image. 

Summary of the Invention 

The deficiencies and limitations of the prior art are overcome by the 
present invention, which provides a method and apparatus for decreasing the 
image acquisition time in digital imaging devices. More particularly, according 
to the present invention, the image acquisition time is reduced by processing 
segments of an entire image and storing the processed image segments onto 
the storage medium. 

According to another aspect of the present invention, at least a portion of 
the image processing pipeline may be divided into a plurality of pipeline stages 
to provide parallel processing of the image segments. 
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According to yet another aspect of the invention, stitching may be 
performed on the processed image segments stored onto the storage medium to 
restore the entire image. 

These and other objects and features of the present invention will be 
5 apparent upon consideration of the following detailed description of preferred 
embodiments thereof, presented in connection with the following drawings in 
which like reference numerals identify like elements throughout. 

Brief Description of the Drawings 

10 Fig. 1 shows the basic elements of a digital imaging device according to 

the present invention; 

Fig. 2 shows another arrangement of the basic elements of the digital 
;S imaging device according to the present invention; 

- n Figs. 3 A and 3B respectively show an original image and image segments 

RJ15 of the original image; 

7n Fig. 4 shows an example of image processing operations performed on 

the image segments shown in Fig. 3B; 
p Fig. 5 shows an example of an image processing pipeline according to an 

jiTs aspect of the present invention; 

C20 Fig. 6 shows an example of an image processing pipeline according to 

p another aspect of the present invention; and 

Fig. 7 shows another arrangement of the basic elements of the digital 
imaging device according to the present invention. 

25 Detailed Description of the Invention 

Referring to Fig. 1, the basic elements of a digital imaging device 100 
include a lens 8 and an image sensor 10 for obtaining an image and a central 
processing unit (CPU) 12 for processing the image data supplied from the 
image sensor 10. The image sensor 10 may be a CCD, a CMOS sensor, a 

3 

03797.00006 



Super CCD as sold by Fuji Photo Film, or any other suitable image sensing 
device. The CPU 12 controls the processing of the image data based upon 
program data stored in ROM 14. The CPU 12 may perform all of the image 
processing or the device 100 may include co-processors 18, in addition to the 
5 CPU 12, for performing the image processing operations. The device 100 also 
includes RAM 16 for data storage. A storage medium 20 is provided for storing 
the processed image data. The storage medium 20 may be any appropriate 
memory such as a hard disk, floppy disk, flash memory, or any digital 
recording medium as is known in the art. 
10 The image data from the image sensor 10 may be supplied to the CPU 

12, as shown in Fig. 1. In another arrangement, shown in Fig. 2, the image 
data may be supplied to a memory 22 from which the CPU 12 reads the image 
J data for processing. A cache 21 may be arranged at any point or at multiple 
p 1 points in the device 100, as shown in Fig. 2. Image data may be stored in any 
H15 of a number of different formats supported by the digital imaging device 100. 
m Examples of these formats include GIF, JPEG, BMP (Bit Mapped Graphics 
O Format), TIFF, FlashPix, etc. Each of these data formats are well known in the 

□ art and do not require description herein. 

jjji According to an embodiment of the invention, the CPU 12 divides the 

jjo image data into segments, as shown in Figs. 3A and 3B. More particularly, the 

□ CPU 12 receives the image data for the original image 24 shown in Fig. 3 A, and 
identifies the beginning and ending locations of image segments Al-An, as 
shown in Fig. 3B. The image segments may or may not be arranged to overlap. 
In the present example, the CPU 12 and the co-processors 18 perform 

25 conventional image processing of the image segments Al-An, such as color 
interpolation in step SI, white balance in step S2, etc., and conversion of the 
data to a particular format in step Sn-1 such as the JPEG format. Finally, the 
image segments are stored on the storage medium 20 in step Sn. The image 
segments are processed through the image processing pipeline. As the 
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processed image segments arrive at the storage medium 20, they are stored. 
The device 100 may also be arranged to perform some image processing before 
the image is segmented (e.g., excess noise or color separation added to be 
eliminated on stitching, as discussed below). 
5 According to the present invention, the image sensor 10 may output the 

entire image to the CPU 12 or it may output each line to the CPU 12 in a 
bucket-brigade manner. For example, the image sensor 10 may be a CCD 
sensor that operates in a bucket-brigade manner so that the image data is 
transferred from the sensor 10 one line at a time and is stored in a memory 22. 
10 When a line of image data is output from the sensor 10, the line of image data 
previously stored in memory 22 is shifted out of the memory 22 and supplied 
to the CPU 12. Usually, the CCD outputs the image data in segments of pixel 
J width or the image is bucket brigaded in pixel width. Therefore, the CCD 
~f\ sensor outputs the image in image segments that are processed as discussed 
nl5 above. 

§1 The image may be segmented into any appropriate number of segments 

^ that facilitates increased image acquisition speed and that the device is 
o equipped to process. In addition, the image may be segmented vertically, 
S] horizontally, diagonally, in rectangular sectors, or in other types of physical or 
JMZO logical groupings. For example, the system may scan the image to determine 
O the general orientation of the picture elements and process along the general 
direction of the elements. As noted above, the segments may overlap with one 
another. The segmentation of the image may depend upon the length of the 
image processing stages and the level of organization available for processing. 
25 The level of image segmentation may also be determined based upon the 
amount of time each processing step requires and the overhead of moving each 
image segment and stitching it with the other image segments. The digital 
imaging device 100 may be preprogrammed to a particular segmentation level, 
or the device may be provided with a number of default levels. The default 
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levels may be selected by the CPU 12 based upon parameters such as 
resolution, white balance, etc. One of ordinary skill in the art will appreciate 
that there may be many ways to implement the image segmentation. 

An example of the pipeline processing of the image segments is 
5 illustrated in Fig. 5. The image data may be supplied to the cache 21, which 
stores the image data as segments Al, A2, A3 and A4, for example. The cache 
21 may store a table that includes the starting and ending locations for each of 
the data segments of the image data. The image data from the cache 21 is 
supplied to the image processing pipeline at pipeline stage 1 28. Image 
10 processing then begins on the first data segment Al. Data segment Al is 
transferred from pipeline stage 1 28 to pipeline stage n 32 to complete image 
processing of data segment Al. Data segment Al may then be stored on the 
% storage medium 20. Similar processing occurs for each of the remaining data 
CO segments A2, A3 and A4. Data segment Al may be stored onto the storage 
fit 5 medium 20 while image processing continues on other data segments of the 
*Z s image data. Therefore, according to the present invention, image processing 
O may be occurring concurrently with writing to the storage medium 20. The 
□ pipeline processing of the data segments increases the processing speed of the 
digital imaging device 100. More particularly, this pipeline processing of the 
data segments reduces the time from obtaining the image at the image sensor 
q 10 to the time when the image data is available to begin storing on the storage 
medium 20. 

The image data may also be supplied to a screen 36 or a personal 
computer (PC) 34, or any other appropriate device. In addition, those of 
25 ordinary skill in the art will appreciate that the device 100 may include buffers 
to facilitate the pipeline processing operation. 

According to another aspect of the present invention, some or all of the 
image processing pipeline stages may be separated to perform parallel 
processing of the image data segments, as shown in Fig. 6. In this example, 
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one of the pipeline stages, pipeline stage 3 is separated into parallel pipeline 
stages 3A 31A and 3B 3 IB. In this arrangement, different data segments of the 
image are processed in parallel in stages 3A and 3B. One of ordinary skill in 
the art will appreciate than any one or a number of the image processing 
5 pipeline stages may be separated into a plurality of parallel stages to decrease 
overall processing time. The device 100 may include buffers to facilitate the 
processing operation. For example, if a pipeline stage is divided into multiple 
parallel processing stages, and the device 100 allows processed segments to be 
output out of order, a buffer may be provided to enable appropriate ordering of 
10 the processed data. 

Once an image segment has been processed, it is stored on the storage 
medium 20. Since the writing operation begins as soon as the first image data 
j2 segment is received at the storage medium 20, the delay in image acquisition 
Cn due to writing speed is minimized, if not eliminated. When each image 
pjl5 segment is stored onto the storage medium 20, it may be stitched to the 
Jpj previously stored image segment. As is well known in the art, stitching 
U includes combining contiguous or overlapping blocks of data to form a 
□ complete block of data. The data segments stored on the storage medium 20 
; ^ may be stitched together to form the entire image. The stitching order may be 
tf|20 in many orientations and orders, such as from top to bottom, right to left, 
q bottom to top, left to right. However, stitching is usually done in the order that 
the data segments are written to the storage medium 20. In addition, the 
stitching may occur by buffering one data segment, performing the stitching 
operation for the segment, and then writing the data to the storage medium 20. 
25 In another arrangement, the data segments may be written to the storage 
medium 20 as they arrive, and then stitched in place. 

In some cases, the data segments may simply be stored on the storage 
medium 20 with what amounts to a null stitching operation. For example, 
some data formats include boundaries for the data, such as 8x8 or 32x32 pixel 
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boundaries. In these cases, the data segments may be identified along those 
boundaries. Then, upon storage onto the storage medium 20, each data 
segment may simply be stored after the previously stored data segment. 
However, in general, data processing, such as data compression, usually 
5 involves interaction or dependency between pixels of the image data. While it 
may not be necessary to stitch every time a data segment is stored onto the 
storage medium 20, if the pixels in one data segment are dependent upon 
pixels in another data segment, then stitching will likely be required. 

Stitching may provide an opportunity to modify image data based upon 
10 total image data. More particularly, stitching may involve making changes or 
modifications to other data segments beyond the two data segments that are 
being stitched together. For example, a particular parameter may be set after 

^ analyzing one data segment. It may be determined later that the initial setting 
for the parameter, based on the single segment, is not appropriate for the 

Ft! 5 entire image. Stitching allows for the possibility of going back and making 

jjfjj modifications to the image data. 

u When the data segments are overlapping, processing occurs using pixels 

o that extend beyond the particular data segment being processed. Once the 
Jrj data segment is processed, any duplicate image data may be discarded. 
^20 Processing overlapping data segments may improve the overall performance of 
Q the imaging device 100 by minimizing the amount of modification required 
upon stitching. More particularly, initial processing decisions for the data 
segment may be more accurate since each data segment is processed using 
information from adjacent image segments providing a better perspective of the 
25 image. 

In some cases, the resulting image file may require a header or some 
other information to be added or updated based on the complete image. This 
may be accomplished by "patching" the image file after all of the image data 
segments have been written onto the storage medium 20. For example, some 
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data files include information at the very beginning of the file such as filename, 
when the file was created, how many bytes are in the file, etc. Some 
information may not be known until the last data segment is stored onto the 
storage medium 20, such as the number of bytes. According to the present 
5 invention, this information may be added at the end of the storage process, or 
may be updated every time a new data segment is stored or at some other 
appropriate time. 

Referring to Fig. 7, another embodiment of the invention is shown. 
According to this embodiment, the image data from the image sensor 10 is 
10 stored in memory 38 and the CPU 12 and each of the co-processors 18, for 
example, read and process different image segments and then store the 
processed image segments onto the storage medium 20. This arrangement 
may be more efficient than that shown in Figs. 1 and 2 when there is no 
8 1 specific hardware architecture to support multiple processors performing the 
fii 5 pipeline processing of the data segments. The device 100 may be arranged so 
§» that a buffer that receives data from the image sensor 10 is used throughout 
D the image pipeline processing operation; that is, the same buffer may be re- 
□ used throughout the processing operation. 

£[* If, for some reason, processing cannot continue in the device 100 shown 

C&O in Fig. 7, the pipeline may be stalled until processing can be resumed. For 
example, if processed data is output out of order, the pipeline may be stalled 
until the necessary data segments are available for continued processing. 
Alternatively, buffers may be provided to allow the pipeline to continue 
processing when data is output out of order. One of ordinary skill in the art 
25 will understand that the device 100 may be arranged in any number of 
architectures to support processing of the image segments. 

While particular embodiments of the invention have been shown and 
described, it is recognized that various modifications thereof will occur to those 
skilled in the art without departing from the spirit and scope of the invention. 
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For example , the invention may be implemented using data schema or web 
pages rather than the directory structure described herein. The described 
embodiments are to be considered in all respects only as illustrative and not 
restrictive. Therefore, the scope of the herein-described invention shall be 
limited solely by the claims appended hereto. 
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What is claimed is: 



1. A method for decreasing image acquisition time in a digital image 
device, comprising: 

capturing an image; 

dividing the image into a plurality of image segments; 

performing image processing on each of the plurality of image segments; 

and 

storing each of the plurality of image segments on the storage medium. 

2. The method according to claim 1, wherein the performing step 
comprises performing image processing on each of the plurality of image 
segments sequentially. 

3. The method according to claim 2, wherein the storing step comprises 
beginning storage of each of the plurality of image segments as soon as a first 
one of the plurality of image segments arrives at the storage medium. 

4. The method according to claim 3, wherein the performing step and 
the storing step are performed simultaneously on at least two of the plurality of 
image segments. 
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5. The method according to claim 3, wherein the dividing step 
comprises: 

dividing the image into a plurality of image segments that overlap one 
another. 

5 

6. The method according to claim 1, further comprising: 

stitching the plurality of image segments together to restore the image 
after the performing step. 

/5l0 7. The method according to claim 6, wherein the stitching step 

SJ comprises: 

JP stitching the plurality of image segments together sequentially following 

~ the performing step. 

[Jil5 8. The method according to claim 6, wherein the performing step 

o comprises: 

performing at least a portion of the image processing in at least two 
parallel image processing stages. 

20 9. The method according to claim 1, wherein the performing step 

comprises: 
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performing at least a portion of the image processing in at least two 
parallel image processing stages. 

10. A computer-readable medium computer-executable instructions for 
performing the steps comprising: 

capturing an image with a digital image device; 

dividing the image into a plurality of image segments; 

performing image processing on each of the plurality of image segments; 

and 

storing each of the plurality of image segments on the storage medium. 

11. The computer-readable medium according to claim 10, having 
further computer-executable instructions for performing the performing step 
comprising performing image processing on each of the plurality of image 
segments sequentially. 

12. The computer-readable medium according to claim 1 1, having 
further computer-executable instructions for performing the storing step 
comprising beginning storage of each of the plurality of image segments as 
soon as a first one of the plurality of image segments arrives at the storage 
medium. 
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13. The computer-readable medium according to claim 12, having 
further computer-executable instructions for performing the performing step 
and the storing step simultaneously on different ones of the plurality of image 
segments. 

14. The computer-readable medium according to claim 12, having 
further computer-executable instructions for performing the dividing step 
comprising: 

dividing the image into a plurality of image segments that overlap one 
another. 

15. The computer-readable medium according to claim 10, having 
further computer-executable instructions comprising: 

stitching the plurality of image segments together to restore the image 
after the performing step. 

16. The computer-readable medium according to claim 15, having 
further computer-executable instructions for performing the stitching step 
comprising: 

stitching the plurality of image segments together sequentially following 
the performing step. 
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17. The computer-readable medium according to claim 15, having 
further computer-executable instructions for performing the performing step 
comprising: 

performing at least a portion of the image processing in at least two 
parallel image processing stages. 

18. The computer-readable medium according to claim 10, having 
further computer-executable instructions for performing the performing step 
comprising: 

performing at least a portion of the image processing in at least two 
parallel image processing stages. 

19. An apparatus for decreasing image acquisition time in a digital 
image device, comprising: 

an image sensor that captures an image; 

a controller that divides the image into a plurality of image segments and 
performs image processing of each of the plurality of image segments and 
outputs processed image segments; and 

a storage medium that stores each of the processed image segments. 
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20. The apparatus according to claim 19, wherein the controller is 
arranged to perform the image processing of the plurality of image segments 
sequentially. 



5 21. The apparatus according to claim 20, wherein the controller is 

arranged to store each of the processed data segments as each of the processed 
data segments arrives at the storage medium. 

22. The apparatus according to claim 19, wherein the controller is 
yiO arranged to stitch the processed image segments together to restore the image. 

|: 23. The apparatus according to claim 22, wherein the controller is 

0 arranged to perform at least a portion of the image processing in at least two 
y parallel image processing stages. 

□ 24. The method according to claim 1, further comprising: 

storing image file information on the storage medium, wherein the image 
file information corresponds to the plurality of image segments for an image 
stored on the storage medium; and 
20 updating the image file information that has been affected by the image 

processing performed on any one of the plurality of image segments. 

16 

03797.00006 



25. The method according to claim 24, further comprising: 
modifying at least one of the plurality of image segments stored on the 

storage medium that has been affected by the image processing performed on 
subsequent ones of the plurality of image segments stored on the storage 
medium. 

26. The computer-readable medium according to claim 10, further 
comprising computer-executable instructions for performing the steps of: 

storing image file information on the storage medium, wherein the image 
file information corresponds to the plurality of image segments for an image 
stored on the storage medium; and 

updating the image file information that has been affected by the image 
processing performed on any one of the plurality of image segments. 

27. The computer-readable medium according to claim 26, further 
comprising computer- executable instructions for performing the step of: 

modifying at least one of the plurality of image segments stored on the 
storage medium that has been affected by the image processing performed on 
subsequent ones of the plurality of image segments stored on the storage 
medium. 
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Abstract of the Invention 

A method and apparatus for reducing the image acquisition time in a 
digital imaging device by processing segments of the image through an image 
processing pipeline and storing the processed image segments. Processing 
segments of the image allows the digital imaging device to write processed data 
segments to a storage medium while other image segments are being 
processed. Any portion of the image processing pipeline may be divided to 
perform parallel processing on the image segments to increase processing 
speed. 
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Date of Filing 
(Day, Month, Year) j ; 


Status □ Patented^ 
Pending, Abandoned 
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Power of Attorney 

And we hereby appoint, both j ointly and severally, as our attorneys with full power of substitution and revocation, to prosecute 
Ms application and to transact all business in the Patent and Trademark Office connected herewith the following at 
attorneys and agents, their registration numbers being listed after their names: 



ALTHERR, Robert F. 
BANNER, Donald W 
BANNER, MarkT. 
BANN|R, Pamela I. 
BECKER, Matthew P. 
BECKETT, William W. 
BERGHAMMER, Joseph J. 
BODNER, Jordan 
BUROW, Scott A. 
CALLAHAN, James V. 
CHANG, Steve S 
COHAN, Gregory J. 
COOPERMAN, Marc S. 
CURT3N, Joseph P. 
DeMOOR, Laura J. 
EVANS, Thomas L. 
FEDOROCHKO, Gary D. 
OSSHER, Daniel E. 
#SHER, William J. 
QQLEMBOCKI, Christopher R. 
H|ANLON ; Brian E. 
jfMEMMENDINGER, Lisa M. 
jJONG, Patricia E. 



31,810 


HOSCHEIT, Dale H. 


19,090 


NIEGOWSKI, James A. 


28,331 


17,037 


HYMEL, Lin J. 


45,414 


PATEL, BmalJ. 


42,065 


29,888 


IWANICKI, John P. 


34,628 


PATHAK, Ajay S. 


38,266 


33,644 


JACKSON, Thomas H. 


29,808 


PAYNE, Stephens. 


35,316 


45,824 


KAGAN, Sarah A. 


32,141 


PETERSON, Thomas L. 


30,969 


18,262 


KATZ, Roberts. 


36,402 


POTENZA, Joseph M. 


28,175 


46,057 


KLEIN, William J. 


43,719 


PRATT, Thomas K. 


37,210 


42,338 


KRAUSE, Joseph P. 


32,578 


RENK, Christopher J. 


33,761 


42,373 


LINEK, Ernest V 


29,822 


RESIS, Robert H. 


32,168 


20,095 


MALONE, Dale A. 


32,155 


R1VARD, Paul M. 


43,446 


42,402 


MANNAVA, Ashok K 


45,301 


ROBINSON, Douglas W. 


32,751 


40,959 


MAPLE, Mane-Claire B. 


37,588 


SCHAD, Steven P. 


32,550 


34,143 


MAY, Steven A. 


44,912 


SHIFLEY, Charles W. 


28,042 


34,571 


McDERMOTT, Peter D. 


29,411 


SKERPON, Joseph M 


29,864 


39,654 


McKEE, Christopher L. 


32,384 


STOCKLEY, D. J. 


34,257 


35,805 


McKIE, Edward F. 


17,335 


VAN ES, J. Pieter 


37,746 


35,509 


MEDLOCK, Nina L. 


29,673 


WITCOFF, Sheldon W. 


17,399 


34,162 


MEECE, Timothy C. 


38,553 


WOLFFE, Franklin D. 


19,724 


32,133 


MEEKER, Frederic M. 


35,282 


WOLFFE, Susan A. 


33,568 


38,800 


MILLER, Charles L. 


43,805 


WRIGHT, Bradley C. 


38,061 


40,449 


MITRIUS, Janice V. 


43,808 


SAKO, Katie E. 


32,628 


42,653 


MORENO, Christopher P. 


38,566 


CROUSE, Daniel D. 


32,022 


34,373 


NELSON, Jon O. 


24,566 







"All correspondence and telephone communications should be addressed to: 



Pamela I. Banner 
Banner & Witcoff, Ltd. 
1001 G Street, N.W., 11th Floor 
Washington, D.C. 20001-4597 



Customer Number: 22907 
Tel: (202) 508-9100 
Fax: (202)508-9299 



D We hereby declare that all statements made herein of our own knowledge are true and that all statements made on 

information and belief are believed to be true; and further that these statements were made with the knowledge that willful 
false statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may jeopardize the validity of the application or any patent issuing 
thereon. 




Signature^ 

Fullj^fie of First mvei 

Residence Redmond. WA 98052 



Date 



Jo nathan 



Family Name 



First Given Name Second Given Name 
Citizenship United States 



Post Office Address 14324 NE 84 th Court. Redmond. WA 98052 
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44. 



Signature 

Full Name of Second Invento r ' " ~ ODINAK Gilad 

Family Name First Given Name Second Given Name 

Residence Bellevue.WA 98005 Citizenshi p United States 

Post Office Addres s 1 2342 NE 26 th Place. Bellevue. WA 98005 
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